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INDUSTRIAL AUGMENTED REALITY 

BACKGROUND OF THE INVENTION 

This invention relates to information processing systems and methods in an 
industrial environment. More particularly, the invention relates to systems and 
methods for displaying information for use by human operators in a mobile working 
environment. 

An increasing number of workers today in mobile working environments are 
assisted by hand-held or wearable computer systems. Rather than using computer 
kiosks or workstations at locations throughout the work environment, the hand-held 
or wearable computers allow the worker to move freely about the workspace and 
retrieve information from computer networks accessible at their fingertips, while they 
maintain in proximity to the object of the task. Examples of these include retail 
operations where sales assistants or inventory control clerks carry hand-held 
computers with bar code scanners that can identify products by scanning the bar code 
and then displaying information associated with that product. Another example 
includes car rental return agents who key information into a wearable computer in the 
parking lot of the rental agency when the car is returned, and then print out a receipt 
from a wearable printer to be able to give to a customer standing in the parking lot by 
the rental car. 

While these systems are useful, they have limited capabilities. Hand-held 
computers require the worker to devote one or both hands to the task of inputting 
information into the computer. Wearable computers generally require the use of at 
least one hand for inputting data into a computer, as well as requiring the worker to 
focus his gaze and attention to the hand-held computer rather than on the object or 
task before him. While these wearable computer solutions are an advance over 
stationary kiosks and strategically located catalogs, wall charts or instruction manual 



books accessible to the worker, there is still much room for improvement to free up 
the hands and attention of the worker, to thereby increase the workers' productivity. 

BRIEF SUMMARY OF THE INVENTION 

The present invention addresses the aforementioned needs by providing a 
method and system for viewing data comprising detecting visual markers within view 
of a wearable camera worn by a human worker, and displaying data associated with at 
least one of those visual markers on a wearable display worn by the operator. 

Another aspect of the invention provides a method and system for 
coordinating movement of human workers comprising detecting of one or more visual 
markers within view of a camera worn by the worker, and determining the location of 
the worker from the known location of the visual marker within view of the camera. 

BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS 

Figure 1 is a system in accordance with one embodiment of the invention. 

DETAILED DESCRIPTION OF THE INVENTION 

Many physical tasks in an industrial work environment can be made more 
efficient by providing workers with more data about their tasks and their 
environment. These tasks may include such things as order picking for factory floor 
activities. This invention makes such tasks more efficient by equipping workers with 
a small wearable display and a small wearable camera. Preferably both the wearable 
display and wearable camera are mounted on a pair eyeglasses or goggles. Preferably 
the camera and display are connected to a small wearable computer. Throughout the 
work environment, low cost visual markers may be strategically placed to identify 
locations or objects in proximity to the markers. When the visual marker is within the 
field of view of the camera, the wearable computer system can "see 5 ' and identify the 
visual marker. Once identified, the wearable display can be used to give the worker 
instructions or information regarding the identified object or location. 



Figure 1 depicts a typical system in accordance with one embodiment of the 
invention. A pair of eyeglasses 10 includes a wearable see-through display 12 with a 
wearable camera 14 both of which are attached to the eyeglasses 10. A headband 16 
assists with keeping the glasses on the worker's head. The wearable camera 14 is 
positioned to detect visual markers 18 within the field of view of the worker. 

The camera and display are preferably connected through cables 20 and 22 to 
a wearable computer 24. However, technological developments may permit the 
connections to be wireless, obviating the need for cables. The wearable computer 24 
includes a belt 26 for strapping it to the waist or an appendage (e.g., arm or leg) of a 
worker. Preferably, the wearable computer 24 includes a wireless transmitter and 
antenna 28 for wireless connectivity to a computer network 30 in the industrial 
environment. 

Also connected with the network 30 are a network server 32, a processor 34, 
and several databases. The databases may include, but are not limited to, a visual 
marker library 36, visual marker location maps 38, production and operation 
reference manuals 40, and employee preference and profile library 40. 

A typical system as described above may include several components that are 
presently commercially available. The wearable display 12 may be a see-through or 
see-around micro-optical display, such as eyeglass displays commercially available 
form Microoptical Corp., Boston, Mass. USA. The small camera 14 may be attached 
to or integrated with the eyeglass displays. Numerous such cameras are commercially 
available. The wearable computer 24 may be one such as commercially available 
from Xybernaut Corp., Fairfax, Virginia, USA, with a wireless communication 
transceiver, for example using an 802.1 1(b) protocol. The wearable computer 24 or 
the processor 34 preferably includes a visual marker or pattern recognition system 
such as, but not limited to, the ARToolkit developed by and available from the 
University of Washington. 



The system also includes a computer network 30 wirelessly accessible to the 
wearable computer. The computer network includes a processor 34 in 
communication with a database located on memory storage devices in communication 
with a server 32. The databases 36, 38, 40 and 42 store the information or data 
associated with the objects associated with the visual markers and or mapped 
locations of the visual markers. The databases may also store data about the work 
environment, object details, current status and location of the workers, etc. The 
server may also be tied into other data sources, such as fire alarms or deliveries to 
keep track of more information to display to the workers. 

The invention also provides various methods for applying augmented reality in 
an industrial work environment. One embodiment of the invention includes a method 
for viewing data, which includes directing a wearable camera worn by a human 
operator towards one or more labeled objects within a field of view of the operator, 
detecting one or more visual markers within a field of view of the camera with at least 
one of the visual markers associated with and proximate to each of the labeled 
objects, selecting data from a memory storage with the data being associated with one 
of the objects associated with one of the visual markers, and then displaying the data 
on a wearable display worn by the operator. 

Preferably, the data is selected by identifying which of the visual markers is 
located within a predetermined zone of the field of view of the camera for a 
predetermined amount of time of between about 0.02 seconds and 0.5 seconds. The 
zone is preferably the central 50% of the field of view of the camera. 

Preferably, the data is displayed on a see-through display such that the data are 
superimposed on a real image seen in the field of view of the operator. More 
preferably, the display is a heads-up see-through display attached to a pair of 
eyeglasses or goggles worn by the operator on which additional data associated with 
the object is displayed in response to a request sent by the operator. The request may 
be an electric signal generated by a voice command from the operator or by a physical 



touch by the operator, or the request may be a focused gaze upon the object for an 
extended period of time. Depending on the job activity, the "query" gaze may be 
between 1 and 2 second of steady gaze for fast-paced activity, such as working in a 
warehouse, to 5 seconds or longer for slow-paced activity, such as working on a small 
parts assembly bench, where a worker's gaze does not change much. 

Additionally, the method preferably includes processing the image of the 
detected visual marker to determine a unique identifier associated with the marker, 
and transmitting the unique identifier to a processor for selecting the data appropriate 
for the workers present task based on contextual clues observed either through the 
visual markers presently in view or recently viewed. Preferably, the image processing 
and pattern recognition is conducted by the werable computer. This processing may 
also be done by a workstation computer on the network, with the wearable computer 
merely transmitting the images viewed through the camera. 

In addition, visual markers placed on stationary objects such as walls or 
stationary equipment allow the computer system to determine where the worker is 
physically located whenever such marker is "seen" by the system. This allows for 
efficient tracking of workers or operators and efficient routing of tasks. 

This invention, thereby, provides for the coordinating of movement of workers 
in a work environment. For example, in a warehouse environment, containers and 
walls can be marked with visual markers. As workers walk around the warehouse, 
the system can track where they are and where they are going. When a worker is 
needed in a specific area, the system can identify the nearest worker and provide them 
with instructions on their wearable display to go to the point of need. When the 
worker arrives and looks at a specific box or object, the system can identify that 
object as the correct object of interest, and provide additional instructions for the 
worker to follow with regard to that object. 

Accordingly, a second embodiment of this invention includes a method for 
coordinating the movement of human workers in a working environment comprising 
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directing a wearable camera worn by a human operator towards a one or more labeled 
objects within a field of view of the operator, and detecting one or more visual 
markers within a field of view of the camera with at least one of the visual markers 
associated with and proximate to each of the labeled objects. Further, the method 
includes processing the image of the visual markers and determining the unique 
identifier associated with each marker, obtaining the physical location of the markers 
maintained in a memory storage; and determining the location of the operator from 
the locations of the markers within the field of view of the camera. 

In addition, the method may preferably include tracking the movement of the 
operator by monitoring the change of the location of the markers within the field of 
view of the camera. The movement of the worker is preferably coordinated by 
displaying data on a see-through display such that the data are superimposed on a real 
image seen in the field of view of the operator with the data including instructions 
directing the operator to a new location. 

The various embodiments of this invention are not limited to a factory floor 
environment, but can be applied to a wide variety of industrial work places. Similar 
scenarios can be described in situations such as hospitals, factories, oil fields, 
chemical plants or anywhere that are physical tasks that require detailed knowledge to 
make the task safer and/or more efficient. 

This invention allows the operator or worker to interact with the work 
environment in a very natural manner. Preferably the wearable camera is focused 
within the same field of view as the operator when focusing straight ahead in a work 
activity. In this way the system can see the objects that the operator is focusing on. 
The system is thus programmed to respond to objects that are within the field of view 
of that camera for at least a predetermined amount of time. If a person stares at an 
object, that usually means they are trying to obtain more information about the object. 
Naturally, in response, the system within identify the object of regard, retrieve 
information about that object from a database, and display that information on the 



wearable display of the operator. The longer that the operator gazes upon the object 
the system may optionally be programmed to provide additional information 
regarding the object. Alternatively, the system may be responsive to voice command 
of the operator or mechanical inputs from the touch of the operator to provide 
additional information upon request of the operator. 

The system can also be programmed to be responsive to the personal 
preferences and other personal information maintained in a profile associated with the 
operator. For example, a worker that has special skills and responsibilities as a 
mechanical maintenance worker would receive information relating to the repair of an 
object within his view. On the other hand, a person whose job responsibilities involve 
the operation or manipulation of objects, such as an assembly line operator, will 
receive information on their display associated with the operation of an object within 
their field of view. Likewise, a firefighter staring at the same object may see 
information about safety precaution or about the volatility of the contents of the 
object. 

Visual based markers are preferred for the system because they are extremely 
low cost. The visual markers can be bar codes, dot matrix codes, or other 
recognizable patterns, such as letters and numbers. While it is possible to use other 
type of non-visual markers such as active or passive radio frequency tags, such tags 
can be expensive. Also such RF tags usually only give general proximity 
information. Visual markers can be located actually on objects. For many 
applications this type of tracking and interaction can be more powerful and/or cost 
effective. 

Depending on the work environment and commonality of objects seen therein, 
more complex visual pattern recognition systems can be incorporated into this 
invention. For example, visual quality control inspection systems rely upon pattern 
recognitions of common elements by comparison with template pattern of such 
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elements. Also, pick and place machines can detect objects out of location by 
comparison of objects with a database of commonly seen objects. 

While this invention has been shown and described in connection with the 
preferred embodiments, it is apparent that certain changes and modifications, in 
addition to those mentioned above may be made from the basic features of this 
invention. In addition, there are many different types of computer software and 
hardware that may be utilized in practicing the invention, and the invention is not 
limited to the examples described above. Accordingly, it is the intention of the 
applicants to protect all variations and modifications within the valid scope of the 
present invention. It is intended that the invention be defined by the following claims, 
including all equivalents. 



